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SYSTEM AND METHOD FOR IMPORTING PERFORMANCE DATA 
INTO A PERFORMANCE EVALUATION SYSTEM 

RELATED APPLICATIONS 

This application claims the benefit of U.S. 
O Provisional Application Serial No. 60/084, 794, entitled 

"METHOD AND SYSTEM FOR DETERMINING THE PERFORMANCE OF A 
5 CALL CENTER AGENT, " (Attorney's Docket 019143.0137), filed 

M, May 8, 1998, which is hereby incorporated by reference. 

^ This application is related to copending U.S. Patent 

jyj . 

* Application Serial No. Oj7 U0 S IPG , entitled "SYSTEM AND 

P. METHOD FOR DEFINING THE ORGANIZATIONAL STRUCTURE OF AN 

Q. 10 ENTERPRISE IN A PERFORMANCE EVALUATION SYSTEM" (Attorney's 

$t Docket 019143.0269); copending U.S. Patent Application 

ffi" (X Serial No. Cfjf l(Q t entitled "SYSTEM AND METHOD FOR 

GENERATING AN EVALUATION IN A PERFORMANCE EVALUATION 
SYSTEM" (Attorney's Docket 019143.0270); copending U.S. 
^V-15 Patent Application Serial No. 0?/ \\Q t (Of , entitled "SYSTEM 

AND METHOD FOR PROVIDING ACCESS PRIVILEGES FOR USERS IN A 
PERFORMANCE EVALUATION SYSTEM" (Attorney's Docket 
019143.0271); and copending U.S. Patent Application Serial 
No. 01 /llO i (01 , entitled "SYSTEM AND METHOD FOR GENERATING 
20 RESULTS IN A PERFORMANCE EVALUATION SYSTEM" (Attorney's 

Docket 019143.0273) . 

TECHNICAL FIELD OF THE INVENTION 

This invention relates generally to evaluation tools, 
25 and more particularly to a system and method for importing 

performance data into a performance evaluation system. 
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BACKGROUND OF THE INVENTION 

Evaluation tools are used to gather and record 
productivity and quality statistics for a business or other 
organization. Analysis of such information allows 

5 inefficiencies and other problems of the business to be 

identified and corrected. In this way, business 

performance is maximized. 

Traditionally, evaluation tools are implemented on 
stand-alone systems that are unable to effectively 
10 communicate with other resources in a network environment. 

Other problems include the use of predefined organizational 
structures that cannot be modified to match the 
E! organizational structure of a business. In addition, 

privileges, evaluations, and reports are generally hard- 
H 15 coded for each evaluation tool. Such customization leads 

yj 

* to high implementation and administration cost. 
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SUMMARY OF THE INVENTION 

In accordance with the present invention, a 
performance evaluation system is provided that 
substantially eliminates or reduces disadvantages or 
5 problems associated with the previously developed systems . 

In particular, the present invention provides a performance 
evaluation system in which users may define configurations 
for importing performance data . into the performance 
evaluation system. 
10 In one embodiment of the present invention, a data 

O file is imported into the performance evaluation system by 

IM, storing a plurality of user-defined data elements. A user- 

•tT defined configuration is stored for the data file. The 

Q 

configuration is operable to identify a data item in the 
15 data file and to map the data item to a data element in the 

s=r=? 

£ performance evaluation system. 

More specifically, in accordance with one embodiment 
Q of the present invention, the configuration associates the 

^1 data item with a member based on a member name or code. 

> •'■ _". 

jo 20 The configuration may also identify a name, type and format 

of the data item. The data item may be identified based on 

its position in a delimited file. 

Technical advantages of the present invention include 

providing an improved performance evaluation system. In 
25 particular, the performance evaluation system allows users 

to define configurations for importing external 

productivity data into the performance evaluation system. 

As a result, data can be imported from a wide variety of 

external devices. In this way, the evaluation process is 
30 streamlined by having data in a centralized location for 

data analysis. 

Other technical advantages will be readily apparent to 

one skilled in the art from the following figures, 

description, and claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present 
invention and its advantages, reference is now made to the 
following description taken in conjunction with the 
5 accompanying drawings in which: 

FIGURE 1 is a block diagram illustrating a performance 
evaluation system in accordance with one embodiment of the 
present invention; 

FIGURE 2 is a block diagram illustrating an example 
0 call center using the performance evaluation system of 

FIGURE 1; 

FIGURE 3 illustrates details of the organization 
tables of FIGURE 1 in accordance with one embodiment of the 
present invention; 
5 FIGURE 4 illustrates details of the privilege tables 

of FIGURE 1 in accordance with one embodiment of the 
present invention; 

FIGURE 5 illustrates details of the plan tables of 
FIGURE 1 in accordance with one embodiment of the present 
0 invention; 

FIGURE 6 illustrates details of the productivity 
tables of FIGURE 1 in accordance with one embodiment of the 
present invention; 

FIGURE 7 illustrates details of the data import tables 
5 of FIGURE 1 in accordance with one embodiment of the 

present invention; 

FIGURE 8 illustrates details of an evaluation file 
stored in the evaluation tables of FIGURE 1 in accordance 
with one embodiment of the present invention; and 
0 FIGURE 9 illustrates details of the filter tables of 

FIGURE 1 in accordance with one embodiment of the present 
invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

FIGURE 1 illustrates a performance evaluation system 
10 in accordance with one embodiment of the present 
invention. As described in more detail below, the 
5 performance evaluation system 10 uses productivity and 

quality data to evaluate the performance of an individual, 
group, process or other suitable type of item or operation. 

Referring to FIGURE 1, the performance evaluation 
system 10 is deployed on a three-tiered architecture. The 

10 three-tiered architecture includes client space 12, server 

application space 14 and database space 16. The client 
space 12 is implemented on a client platform 18 such as a 
work station, personal computer or other device capable of 
communicating between a user and a server. The server 

15 application and database spaces 14 and 16 are implemented 

on a server platform 20 such as a personal computer or 
other device capable of being remotely accessed over a 
network. 

The client and server platforms 18 and 20 are 

20 connected by a network 22. The network 22 may the Internet 

A 

or other wide area network (WAN) , an enterprise intranet or 
other a local area network (LAN) , or other suitable type of 
link capable of communicating data between the client and 
server platforms 18 and 20. 

25 The client space 12 includes a web-browser 30 having 

a graphical user interface (GUI) 32. The graphical user 
interface 32 displays web pages 34 downloaded over the 
network 22 from the server application, space 14. The web 
O^. pages 34 may^use HTML^ or ^Acti - vc X cont - rols 3 &. The Active 

30 X controls ^3%^ provide a rich interface 32 with superior 

scripting control. 

The server application space 14 includes a server 

engine 40, business objects 42, reports 44, charts 46, and 

^ HO ,0 

a data importer 48. The server engihe^-4£- includes active 

35 server pages (ASP) 50 that include server-side components 
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and scripting. The server-side components are specialized 
Active X components executed prior to delivery of web pages 
to the client space 12. The server-side scripting accesses 
the interfaces of the server-side components. Use of the 

5 active server pages 50 allows the web pages to be updated 

in response to changing system and database settings. 

The business objects 42 call stored procedures in the 
database space 16 and perform preprocessing and expressions 
processing functions. In a particular embodiment, the 

0 business objects 42 include an organization object, a plan 

object, an evaluation object, an import service object, and 
a results object. In this embodiment, the organization 
object provides an interface for basic organizational 
functionality. The organization object also provides an 

5 interface to a class of services table giving the client 

space 12 access to allowed functions. The plan object 
provides an interface for the manipulation and use of 
evaluation guidelines. The evaluation object provides 
interfaces for performing evaluations. The import service 

0 object schedules and retrieves external data and places it 

into a delimited file. The import service object also 
provides a basic interface to the configuration settings. 
The results object receives information from the client 
space 12 for selecting reports based on the requested 

5 filters. Each of the objects may comprise one or more 

objects . 

The reports 44 may include both detail and summary 
reports. The detail reports provide information for 
members and elements of each member. In a particular 

0 embodiment, the detail reports include a productivity and 

quality report, a quality evaluation report, a productivity 
analysis report, a quality question report, a productivity 
element report, and a detail evaluation report. The 
productivity and quality report presents actual scores, 

5 maximum scores, and percentages of maximum for both quality 
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and productivity by an evaluation date for each member with 
group averages for all hierarchical levels. The quality 
evaluation report presents notes, actual scores, maximum 
score, and percentages of maximum for each question of an 
5 evaluation with subtotals by performance areas, evaluation, 

member, and any additional hierarchical levels. The 
productivity analysis report presents actual scores, 
maximum score, and percentage of maximum for each equation 
of productivity analysis with subtotals by member and by 

10 any additional hierarchical levels. The quality and 

question report presents actual scores for each question 
along with the total actual score, maximum score, and 
percentage of maximum for each performance area with 
subtotals by evaluation, member, and any additional 

15 hierarchical levels. The productivity element report 

presents actual data along with the equation-produced 
actual score, maximum score, and percentage of maximum for 
each element of productivity data with subtotals by member 
and any additional hierarchical levels. The detail 

20 evaluation report presents evaluation date, time, and 

status of all evaluations for a member subtotal by status, 
date, member, and evaluator. 

The summary reports provide average scores for a 
particular member or level. In a particular embodiment, 

25 the summary reports include a productivity and quality 

report, a quality evaluation report, a productivity 
analysis report, a quality question report and a 
productivity and quality comparison report. The 
productivity and quality report presents average scores for 

30 both quality and productivity for each member with group 

averages for all hierarchical levels. The quality 
evaluation report presents average scores by performance 
areas, evaluation, member, and any additional hierarchical 
levels. The productivity analysis report presents average 

35 scores for each equation of productivity data with 
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subtotals by member and any additional hierarchical levels. 
The quality question report presents the total score for 
each question and the percentage of maximum scores for each 
performance area. The productivity and quality comparison 
5 report presents average scores, maximum scores and 

percentage of maximum for both quality and productivity by 
member with comparison averages for a chosen hierarchical 
level. Additional detail and summary reports can be 
created by selecting various categories of information. 

10 The charts 4 6 display graphic and textual information 

useful. in determining the overall performance of a member 
and identifying ways to improve that performance. In a 
particular embodiment, the charts 4 6 include sets of 
productivity and quality charts, productivity charts and 

15 quality charts. The productivity and quality charts plot 

productivity and quality scores for date, member, users and 
hierarchical levels. The productivity charts present 
productivity data plotted against time for members, users, 
and other hierarchical levels. The quality charts present 

20 quality data plotted against time for members, users, 

questions and hierarchical levels. 

The data importer 48 imports productivity data from 
external sources such as a telephony switch. The data 
importer 48 includes preprocessors 52 and a work area 54. 

25 The preprocessors 52 are tools used to create an 

instruction file for interpreting the information in a data 
file. The instruction files are operable to parse data 
files. In one embodiment, the preprocessors 52 include 
Nortel Datastream, Rockwell Spectrum IMS, and Monarch. The 

30 Nortel Datastream preprocessor is used for converting 

Nortel Datastream report information into a delimited 
format. The Rockwell Spectrum IMS preprocessor is used for 
converting Rockwell report m information. The Monarch 
preprocessor can be used to create custom preprocessors. 

35 Accordingly, the performance evaluation system 10 can be 
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configured to import any type of suitable data file 56 from 
an external device 58. 

The preprocessors 52 create a delimited file 60 from 
a data file 56. In the delimited file 60, columns of data 
5 are each separated by a delimiter character such as a 

comma/ hyphen, colon, and the like. The delimiter acts as 
reference point telling the data importer 48 where one 
column of data stops and a new column starts. If the data 
file 56 is already in a delimited format, a preprocessor 52 
10 is not used. 

O The delimited file 60 is a temporary file stored in 

jM= the work area 54. As described in more detail below, the 

delimited file 60 is moved to the database space 16 and the 
jM 5 ' productivity data mapped into the database based on the 

H 15 configuration defined in the database. Accordingly, the 

yj 

js evaluation process is streamlined by having data in a 

?i central location for analysis. 

O The database space 16 includes a database manager 70 

La 

and a database 72. The database manager 70 calls stored 
|g 20 procedures 80 to access the database 72. The stored 

procedures 80 are a precompiled collection of SQL 
statements and optional control-of-f low statements stored 
under a name and processed as a unit. The stored 
procedures 80 are stored within a database 72, can be 

25 executed with one call from the database manager 70, and 

allow user-declared variables, conditional executions, and 
other programming features. 

In addition to the stored procedures 80, the database 
72 includes organization tables 82, privileges tables 84, 

30 plans tables 86, data import tables 88, productivity tables 

90, evaluation tables 92 and filter tables 94. As 
described in more detail below, the organization tables 82 
allow an enterprise to set up the performance evaluation 
system 10 to correspond to its organizational structure. 

35 The privileges tables 84 store user access privileges based 
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on views of the organizational structure and on class of 
services. The plan tables 86 store questions, performance 
areas, and guidelines for generating performance 
evaluations in the performance evaluation system 10. The 

5 data import tables 88 store configurations for importing 

productivity data into the performance evaluation system 
10. The productivity tables 90 store productivity data for 
members of the performance evaluation system 10. The 
evaluation tables 92 store responses and scores for 

0 completed evaluations. The filter tables 94 store filters 

for sorting data and displaying results in the reports 44 
and charts 46 . 

In a particular embodiment, the client and server 
platforms 18 and 20 are each a personal computer connected 

5 via the Internet. In this embodiment, the client and 

server 18 and 2 0 may each operate using MICROSOFT WINDOWS 
NT version 4.0 that provides TCP/IP connectivity over the 
Internet. The web-browser 30 may be Internet Explorer 3.2x 
or higher. The server engine 40 may be Internet 

0 Information Server 3.0 or higher. The database may be a 

relational database and the database manager 70 an SQL 
Server 6.5. 

The web-base architecture of the performance 
evaluation system 10 allows performance evaluation 

5 methodology to be standardized throughout an enterprise. 

In addition, users can take advantage of the Internet or 
other network to remotely access the performance evaluation 
system and complete member evaluations. Members can also 
remotely track their progress. 

0 FIGURE 2 illustrates an exemplary call center 100 

using the performance evaluation system 10. Although the 
performance evaluation system 10 will be described in 
connection with the exemplary call center 100, it will be 
understood that the performance evaluation system 10 may be 
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used for other types of enterprises, including other types 
of companies, businesses, groups, and organizations. 

Referring to FIGURE 2, the call center 100 includes a 
server 102 connected to a plurality of clients 104 through 
a local area network (LAN) 106. The call center 100 
includes a system administrator (user 1), a service manager 
(user 10), a technical manager (user 20), a sales manager 
(user 30), a product A manager (user 31), and a product B 
manager (user 35) . Service agents (members 11 and 12) 
report to the service manager. Technical agents (members 
21 and 22) report to the technical manager. Product A 
agents (members 32 and 33) report to the product A manager. 
Product B agents (members 3 6 and 37) report to the product 
B manager. In addition, the product A and B managers 
report to the sales manager. 

FIGURE 3 illustrates details of the organization > 
tables 82. The organization tables 82 allow an enterprise 
to set up the performance evaluation system 10 to fit the «: 
organizational structure of the enterprise. Thus, the 
organization tables 82 may store any number of user-defined 
levels in any user-defined hierarchy. In this way, quality 
and productivity tools are customized for the enterprise. 



include a level table 120, an interlevel assignment table 
122, member tables 124, a user table 126, a user-member 
table 128, a keyword table 130, and a keyword assignment 
table 132. The level table 120 lists all the levels of the 
enterprise by a level ID. The levels are a collection of 
members who can be grouped together. The level IDs may be 
any suitable identifier operable to uniquely identify the 
levels. For the call center 100 of FIGURE 2, the levels 
are the company, the service group, the technical group, 
the sales group, the product A team, and the product B 
team. 



Referring to FIGURE 3, the organization tables 
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0^ The interlevel assignment table 122 assigns flower 

level in the enterprise hierarchy to a higher level in the 
hierarchy. In the interlevel assignment table 122, a first 
field identifies a higher level to which a lower level is 
5 assigned and a second field identifies the lower level. 

The labels "first" and "second" are used to distinguish 
between fields and not to specify a particular order for 
the fields. The lower levels may be assigned to a next 
higher level in the hierarchy or to each higher level in 

10 the hierarchy. The levels are identified in the interlevel 

assignment table 122 by the level IDs. For the call center 
100 of FIGURE 2, the service, technical, and sales group 
levels are assigned to the company level and the product A 
and B team levels are assigned to the sales group level., 

15 The member tables 124 list in a first field all the 

members of the enterprise by a member ID. The members are 
those individuals of the enterprise on which functions of 
the. system 10 are performed. Each member ID may be unique 
codes assigned by the performance evaluation system 10 when 

20 the member is first defined in the system 10. 

£a- In the member -attribute tables 124, a second field 

assigns the member to a level. Additional fields provide 
member attributes, such as first and last name, password, 
comments, and the like. In another embodiment, some or all 

25 of the member attributes may be stored in a separate table. 

For the call center 100 of FIGURE 2, service agents 

(members 11 and 12) are assigned to the service group, 

technical agents (members 21 and 22) are assigned to the 

technical group, product A agents (members 31 and 32) are 

3 0 assigned to the product A team, and so on through product 

£> 

B agent (member 37) assigned to product^ team. 

A 

The user table 126 lists all the users of the 
enterprise by a user ID. The users are people in the 
performance evaluation system 10 that perform the 
35 evaluations and that carry out the various tasks associated 



Docket No. 0flBW3.0272 



PATENT 



13 

with the evaluation process such as defining all of the 
information needed to perform the evaluations. The user 
IDs may be any suitable identifier operable to uniquely 
identify the users. For the call center 100 of FIGURE 2, 

5 the users are the system administrator (user 1), the 

service manager (user 10), the technical manager (user 20), 
the sales manager (user 30) , the product A manager (user 
31), and the product B manager (user 35). 

The users-member table 128 cross-references user IDs 

0 with member IDs for users who are also members in the 

system 10. A user is also a member when the user reports 
to and is evaluated by a user higher in the enterprise 
hierarchy. For the call center 100 of FIGURE 2, users 31 
and 35 report to user 30. Accordingly, users 31 and 35 are 

5 cross-referenced as members in the users-member table 128. 

The keyword table 130 lists all the keywords defined 
in the performance evaluation system 10 by a keyword ID. 
The keywords provide an easy way to classify members and 
other items of the performance evaluation system 10. The 

0 keywords can also be used to filter performance data for 

analysis or printing reports. The keyword IDs may be any 
suitable identifier operable to uniquely identify the 
keywords in the system 10. For the call center 100 of 
FIGURE 2, the keywords may relate to the experience of an 

5 agent or to an agency from which the agent was hired. In 

this way, the performance of the agents may be compared 
based on their experience, the agency from which they were 
hired, and the like. 

The keyword assignment table 132 assigns keywords to 

0 a member. In the keyword assignment table 132, a first 

field identifies a member by a member ID and a second field 
identifies a keyword assigned to the member by a keyword 
ID. One or more keywords may be assigned to each of the 
members. For the call center 100 of FIGURE 2, if the 

5 service agent (member 11) is hired from the AA Agency and 
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has -*e 5 s-s — th an ■ two years' experience, keywords "Temp-AA 
Agency" and f ^2 years experience" are assigned to the 
service agent (member 11) . 

FIGURE 4 illustrates details of the privilege tables 
5 84. The privilege tables 84 assign each user a view and a 

class of services. The view specifies the levels and 
members of an organizational structure to which the user is 
allowed access. The class of services specifies services 
of the performance evaluation system 10 that the user is 
10 allowed to perform. The user has access privileges to 

O perform services within the user's class of services for 

levels and members within the user's view. In this way, 
access privileges may be easily updated and maintained for 
the user in response to changes in allowed services, or 
15 organizational structure. As a result, system 

administration cost is reduced. 

Referring to FIGURE 4, the privilege tables 84 include 
O a user view table 140, a class of service assignment table 

Xf= l 142, and a class of service template table 144. The user 

f@ 20 view table 140 specifies levels and members of the 

organizational structure to which the user is allowed 
access. 

In the user view table 140, a first field identifies 
a user by user ID, a second field identifies a level or 

25 member associated with the user by the level or member ID, 

a third field identifies whether the item in the second 
field is a level or a member, and a fourth field enables 
the stored association between a user and a level or member 
in the user's view. Thus, in this embodiment, each user is 

30 associated with each level in the user's branch of the 

organizational structure and with members that report to 
the user. Each association between a user and a level or 
member may be enabled or disabled in the user's view and 
deselected using column 6.96 . to be excluded from the user's 

35 view. In another embodiment^, the user view table 140 may 



Q 
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include only associations that are included in the user's 
view. In this embodiment , the fourth field is omitted. 

For the call center 100 of FIGURE 2, the system 
administrator (user 1) is associated with each level and 

5 member in the performance evaluation system 10. All of the 

associations for the system administrator are enabled and 
thus included in the system administrator's view to allow 
the system administrator to access each level and member of 
the system. The service manager (user 10) is associated 

0 with each level in the service manager's branch of the 

organizational structure and with members that report to 
the service manager. Thus, the service manager is 
associated with the company and service group levels and 
with the service agents (members 10 and 11) that report, to 

5 the service manager. The associations between the service 

manager, the service level and the service agents are 
enabled to allow the service manager to access the level 
and members to configure and perform performance 
evaluations and other functions for the members and level. 

0 The association with the company level is not enabled and 

thus excluded from the service manager's view. Similarly, 
the product B manager (user 35) is associated with the 
company, sales group, and product A and B team levels and 
with the product B agents (members 36 and 37) that report 

5 to the product B manager. The associations with the 

product B team level and product B agents are enabled and 
included in the product B manager's view. Access to the 
company and the sales group levels are not enabled and thus 
excluded from the product B manager's view. 

0 The user view table 140 may be manually populated by 

the system administrator or other user having suitable 
access privileges or may be generated based on the 
organizational structure defined by the organization tables 
82. In the latter embodiment, a user assignment table is 

5 used to assign each user to a level. Based on that 
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assignment, the organization tables 82 are interrogated to 
determine the levels within the user's branch of the 
organizational structure and the members that report to the 
user. This information is then used to populate the user 
5 view table 140. After the table is populated, associations 

are to be included in the user's view. 

The class of service assignment table 142 assigns a 
class of service template to a user. In the class of 
service assignment table 142, a first field identifies the 

10 user by the user ID and a second field identifies the class 

of service template by a service template ID. The service 
template ID may be any suitable identifier operable to 
uniquely identify the service templates. Use of the 
service templates allows a particular class of services to 

15 be defined and reused for any number of users. .For 

example, for the call center 100 of FIGURE 2, group and 
team templates may be defined and associated with the group 
and team managers, respectively. Accordingly, the 

performance evaluation system 10 is more efficiently 

20 configured and administration cost is reduced. 

The class of service template table 144 specifies the 
class of services for each service template. The class of 
service template table 144 includes a first field 
identifying a service template by the service template ID. 

25 Additional fields are included to specify the services to 

be included in the service template. The class of service 
template table - l§ 4-also includes a field for identifying a 
specific user by the user ID. This is used to generate a 
custom class of services for the user rather than a service 

30 template. 

The services are predefined for the performance 
evaluation system 10. In a particular embodiment, the 
services may include administer productivity values, 
administer productivity data elements, access charting and 

35 reporting and administer filters, administer keywords, 
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administer member information, administer class of service, 
administer evaluation expressions, configure data import 
settings, import data, administer organizational structure, 
perform reviews, add or modify reviews, delete reviews, 
5 administer performance areas and questions, browse my 

reviews, browse my performed reviews, browse all reviews, 
and administer user information. 

FIGURE 5 illustrates details of the plan tables 86. 
The plan tables 86 store questions, performance areas 
10 constructed of a set of questions, and guidelines 

O constructed of a set of performance areas. The questions, 

^ performance areas, and guidelines are used to generate 

H performance evaluations for members of the performance 

a 

jj,. evaluation system 10. The use and sharing of questions by 

H 15 multiple performance areas and the use and sharing of 

~" performance areas by multiple guidelines allows evaluations 

H to be efficiently defined and automatically generated. In 

5 

Q addition, members in a group are evaluated against the same 

*t set of criteria to provide fairness and objectivity. As* a 

0B 20 result, managers can perform consistent, objective 

evaluations efficiently. 

Referring to FIGURE 5, the plan tables include a 
question table 150, an answer types table 152, a 
performance area table 154, a question assignment table 
25 156, an expression table 158, an expression assignment 

table 160, a guideline table 162, and a performance area 
assignment table 164. The question table 150 defines 
questions that will be used in the performance evaluations. 
The question table 150 includes a first field identifying 
30 the question by a question ID, a second field providing a 

title for the question, a third field specifying text for 
the question, and a fourth field assigning a weight to the 
question. The question ID may be any suitable identifier 
operable to uniquely identify the questions. 
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For the call center 100 of FIGURE 2, a first question 
may be directed toward schedule adherence and ask "How 
often does the agent work on the originally assigned 

C\_ shift?" and a second question -iro^ bte directed toward 

5 promptness and ask "How often is the agent late for work?". 

Relative weights are assigned to the question depending on 
their importance to the enterprise. Thus, if promptness is 
more important to the enterprise than schedule adherence, 
the second question will be assigned a greater weight. 

10 The answer types table 152 associates each question 

with a predefined answer type and attributes for that 
answer type. In the answer types table 152, a first field 
identifies the question by the question ID, a second field 
identifies the answer type by an answer type ID, and a 

15 third field identifies the attributes for the answer type. 

The answer type IDs may be any suitable identifier operable 
to uniquely identify the answer types. 

In a particular embodiment, defined answer types 
include yes/no, drop-down lists, horizontal scroll bar, 

20 edit box, and radio button answers. The yes/no answer type 

asks for either a yes or a no response. The drop-down list 
asks for one answer from a list of possible answers. The 
horizontal scroll bar answer type asks for an answer on a 
continuum for an overall rating. The edit box answer type 

25 asks for a numeric answer. The radio buttons ask for one 

answer from a list of possible answers. Attributes for the 
answer types include where appropriate, target values, 
labels, maximum and minimum values, and the like. 

The performance area table 154 lists all the 

30 performance areas in the performance evaluation system 10 

by a performance area ID. Each performance area is a 
grouping of questions that relates to a particular area of 
job performance. The performance area IDs may be any 
suitable identifier operable to uniquely identify the 

35 performance areas. For the call center 100 of FIGURE 2, 
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the performance areas may include on-call greeting and call 
closing. 

The question assignment table 156 assigns questions to 
the performance areas. In the question assignment table 
5 156, a first field identifies a performance area by a 

performance area ID and a second { field identifies a 
question assigned to the performance area by a question ID. 
One or more questions may be assigned to each of the 
performance areas. In addition, questions may be reused 
10 between performance areas. For example, performance areas 

2, 3 and 4 may each include question 3. 

The expression table 158 lists in a first field all 

H= the expressions in the performance evaluation system 10 by 

O 

j^. an expression ID. The expression IDs may be any suitable 

O 15 identifier operable to uniquely identify the expressions. 

^ A second field specifies a mathematical equation for the 

O expression. The equation may contain data elements, 

q numeric values, mathematical operations or mathematical 

functions. In a particular embodiment, the mathematical 
qi 20 operations include addition, subtraction, multiplication 

and division. The mathematical functions include sum, 
average, minimum, maximum, and peak. 

The expression assignment table 160 assigns 
expressions to the perf ormance areas. When associated with 
25 a performance area, the expressions are used to compute a 

productivity score for that performance area. In the 
expression assignment table 160, a first field identifies 
a performance area by a performance area ID and a second 
field identifies an expression assigned to the performance 
30 area by an expression ID. One or more expressions may be 

assigned to each of the performance areas. In addition, an 
expression may be reused between performance areas. For 
example, performance areas 1 and 6 may each include 
expression 1. 
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The guideline table 162 lists all the guidelines in 
the performance evaluation system 10 by a guideline ID. 
Each guideline combines one or more performance areas that 
are of interest for a particular evaluation. The guideline 
5 IDs may be any suitable identifier operable to uniquely 

identify the guidelines. For the call center 100 of 

FIGURE 2, the performance areas may include technical 
group, product A team, product B team, and the like. 

The performance area assignment table 164 assigns 
10 performance areas to the guidelines. In the performance 

O area assignment table 164, a first field identifies a 

m guideline by a guidelines ID and a second field identifies 

H a performance area assigned to the guidelines by a 

O 

1^ performance area ID. One or more performance areas may be 

15 assigned to each of the guidelines. In addition, 

performance areas may be reused between guidelines. For 
the call center 100 of FIGURE 2, for example, the service, 
p technical, product A team, and product B team guidelines 

H may each include performance area 1 . 

g3 20 FIGURE 6 illustrates details of the productivity 

tables 90. The productivity tables 90 store productivity 
data for members of the performance evaluation system 10. 
The productivity data is used by the expressions in the 
performance areas to calculate productivity scores. 

25 Referring to FIGURE 6, the productivity tables 90 

include a data element table 166 and a productivity table 
168. The data element table 166 lists all the data 
elements in the performance evaluation system 10 by a data 
element ID. The data elements are items of information 

30 used to measure the productivity of members in. the 

performance evaluation system 10. The data element IDs may 
be any suitable identifier operable to uniquely identify 
the data elements. 

The productivity table 168 includes a first field 

35 identifying members with a member ID, a second field 
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providing dates for the records, a third field identifying 
the data element for the record by a data element ID, and 
a fourth field storing the value of the data element. 

FIGURE 7 illustrates details of the data import tables 
5 88. The data import tables 88 import productivity data 

into the performance evaluation system 10 from external 
devices and automatically insert it into the production 
table. As a result, the evaluation process is streamlined 
by having data in a centralized location for data analysis. 
10 Referring to FIGURE 7, the data import tables 88 

include configuration tables 170, an import template 
assignment table 172, and an import template table 174. 
H Each configuration table 170 provides information for 

m identifying and mapping data from the delimited file 60 to 

15 the productivity table 90. 

The configuration table 170 includes a first field 

Q identifying a column in the delimited file 60 containing a 

M 

q data item corresponding to a data element, a second field 

^Z' identifying a data name, a third field identifying a data 

0] 20 type, a fourth field identifying a data format, and a fifth 

field mapping the data item to the data element in the 
productivity table 90. In a particular embodiment, the 
data types include identity, date, duration and numeric 
types. An identity column provides identifying information 
25 such as a name or code. The date column provides the date 

of the inf ormation. The duration column shows an amount of 
time for the information. A numeric column contains a 
value. 

The import template assignment table 172 assigns an 
3 0 import template to a configuration. In the import template 

assignment table 172, a first field identifies the 
configuration by a configuration ID and a second field 
identifies the import template by an import template ID. 
The configuration and import template IDs may be any 
35 suitable identifiers operable to uniquely identify the 
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configurations and import templates, respectively* Use of 
the import templates allows a process for imported data to 
be defined and reused for any number of configurations. 
Accordingly, the performance evaluation system 10 is more 
5 efficiently configured and administration cost is reduced. 

The import template table 174 specifies preprocessor 
and other information for importing the data file 56. The 
import template table 174 includes a first field 
identifying the import templates by the import template 
10 IDs. A second field identifies a preprocessor, if any, to 

O be applied to the data file 56 to generate the delimited 

j~l file 60. A third field identifies the delimiter used in 

H the delimited file 60. 

p. In operation, productivity data is imported by 

15 selecting a configuration, selecting users to determine how 

the data will be applied (members assigned to selected 

users receive the productivity data) , specifying how the 

-J 

q date will be determined, selecting the data file 56 to 

H import by selecting a drive and the file name, specifying 

IP 20 how to match productivity data to members (by name or 

code) , and selecting whether or not to replace existing 
productivity information. In response to the import 
request, the data importer 48 retrieves the requested data 
file 56 and if the import template associated with the 
25 configuration requests a preprocessor 52, applies the 

preprocessor 52. Using the delimiter information in the 
import template for the configuration and the mapping 
information in the configuration table, the data importer 
48 inserts the productivity data into the productivity 
30 tables 90 for members in the user's view. 

FIGURE 8 illustrates details of an evaluation stored 
in the evaluation tables 92. As previously described, the 
evaluation is automatically generated based on guidelines, 
performance areas, and questions selected for the 
35 evaluation. 
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Referring to FIGURE 8, the evaluation includes a 
member ID 180 for whom the evaluation was completed, a date 
182 on which the evaluation was completed, and a plurality 
of performance areas 184. The performance areas 184 each 
include one or more questions 186 and their associated 
answer types 188. As the evaluation is completed, 
responses 190 are stored for each question and used to 
calculate a quality score 192 for the performance area 184. 
The quality score 192 is a measure of "goodness" for the 
member 1 s performance . 

The quality score 192 for a performance area 184 is 
calculated based on the relative weight of the questions 
186 in the performance area 184. The score of responses 
190 to all questions 186 in the performance area 184 are 
summed and divided by the sum of the relative weights. 
This produces a quality score 192 for the performance area 
184. Mathematically, this is expressed in the following 
way: 

± [ { ach ' eved ' - ) * relWeight ) 

n 

£ relWeight i 

i=i 



where 

n is the number of applicable questions in the 
performance area 

i ranges over the answered questions in the 
performance area 

achievedi is the number of points the member actually 
earned for question i 

worst-i is the lowest possible score that can be 
assigned to question i 

relWeighti is the relative weight for question i 

After the quality scores 192 have been calculated for 
the performance areas 184 in an evaluation 92, the quality 
scores 192 are summed and divided by the sum of the 
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relative weight for all questions to derive a total 
evaluation quality score 194. Mathematically, this is 
expressed as: 

n 

£ qScore /; 
1=1 

n 

52 relWeighty 



5 where 

n is the number of applicable questions in the 
evaluation 

i ranges over the applicable questions in the 
evaluation 

0 qScore ± is the q-score for question i 

relWeight i is the relative weight for question i 

The performance areas 184 may also each include one or 
more expressions 200 to calculate a productivity score 202. 

5 As previously described, the expression 200 is a user- 

defined formula to calculate a measure of productivity 
against a target score. 

The calculation of productivity score 202 for a 
performance area 184 is a direct computation of the 

0 expression 200 using the productivity data. After the 

productivity score has been calculated for all performance 
areas 184 that have an associated expression 200, the 
performance area productivity scores 202 are tallied to 
derive a total productivity score 204 for the evaluation. 

5 Mathematically, this is expressed as a median score of all 

performance area productivity scores: 

achieved; 
jf\ tar Z et i 

n 



)fl^43.0272 



Docket No. 01^4 3.0272 PATENT 

25 

where 

n is the number of performance areas in the evaluation 
having an associated evaluation expression that can be 
successfully calculated 
5 i ranges over the performance areas in the evaluation 

achievedi is the number of points the member actually 
earned for performance area i 

target ± is the target value for performance area i 

0 In another embodiment, the productivity scores for the 

different performance areas - 3 2 0^ may be weighted based on 
importance. In this embodiment, the overall productivity 
score for the evaluation is calculated by the following 
equation: 

** i 

achieved t 



7TT A tar z et i 
n 



5 where 

n is the number of performance areas in the evaluation 
having an associated evaluation expression that can be 
successfully calculated 

i ranges over the performance areas in the evaluation 
0 w ± is the importance factor associated with performance 

i\s p-score 

'achieved ± is the number of points the member actually 
earned for performance area i 

target ± is the target value for performance area i 

5 

In operation, the user selects the member 180 and an 
existing guideline or one or more performance areas 184 for 
the evaluation. The evaluation is then automatically 
generated. Quality scores are generated based on the 

0 user's responses. Productivity scores are generated based 

on productivity data. 

FIGURE 9 illustrates details of the filter tables 94. 
The filter allows the user to define certain criteria for 
reporting and charting. This criteria is used to sort the 

5 data and display results in the charts and reports. 
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Referring to FIGURE 9, the filter tables 94 include a 
first field identifying the filters with a filter ID. The 
filter ID may be any suitable identifier operable to 
uniquely identify the filters. A second field indicates 

5 whether the filter includes plan criteria. Each category 

of plan criteria is specified by a separate look-up table. 
In a particular embodiment, the categories include 
evaluations, guidelines, performance areas, questions, 
expressions, and evaluators. In this embodiment, an 

0 evaluation filter table associates the filter with 

evaluations, a guideline filter table associates the filter 
with an identified guideline, a performance area filter 
table associates the filter with a specified performance 
area, a question filter table specifies questions for the 

5 filter, an expressions filter table specifies expressions 

for the filter and an evaluators filter table specifies 
evaluators for the filter. Each of these filter's tables 
include a first field identifying the filter with a filter 
ID and a second field identifying the relevant evaluation, 

0 guideline, performance area, question, expression or 

evaluator for the filter. 

Returning to the filter table 94, a set of date fields 
specify date(s) for the filter. The date may be a current 
day, current month, current quarter, current week, current 

5 year, previous day, previous month, previous quarter, 

previous week, previous year, or be otherwise defined by 
the user. A set of data fields specify data for the 
filter. The data may be members, all members, or member 
levels. A set of averages fields specify averages for the 

0 filter. The averages may be daily, weekly, bi-weekly, or 

monthly. A set of sample size fields specify sample size 
for the filter. The sample size allows data to be filtered 
based on any sample size entered by the user. A keyword 
field indicates whether the filter includes any keywords. 
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Keywords are associated with the filter in a separate look- 
up table as previously described for plan criteria. 

Thus, evaluators, guidelines, performance areas, 
questions, expressions, dates, data, averages, keywords, 
5 sample size, and other criteria may be specified for the 

filter. As a result, a user may narrowly define 
information and obtain meaningful results. 

Although the present invention has been described with 
several embodiments, various changes and modifications may 
10 be suggested to one skill in the art. It is intended that 

the present invention encompass such changes and 
modifications as fall within the scope of the appended 
claims . 



